Reinforcement Learning Adaptive PID Controller for an Under-actuated Robot Arm

Under-actuated robot manipulator is a kinematic chain wherein total degree of the freedom of the mechanism is more than actuators. Under actuated manipulators are advantageous from the minimalism viewpoint in robotics where a task is performed with less energy consuming actuations. In fact dynamic of the mechanism is exploited instead of fighting it [1][2]. Moreover, studies on under-actuated can be beneficial in building of fault tolerant mechanisms, as when some joints of a fully actuated manipulator fail, the task can be continued before need for repairing them. Control of under-actuated manipulators is a challenging issue because of their nonlinear characteristics and the lake of global controllability. Fortunately, It was proven that these manipulators have small-time locally controllability on an open subset of their zero velocity section, which allow them to follow any path in this subset [3]. This fact makes adaptive controllers as suitable choice for under-actuated manipulators. Among different controllers, PID controllers are the most popular ones due to their simple implementation and high reliability. Moreover, in most cases model-free methods are available for tuning of PID parameters. As a result PID controllers have been extensively used in industries. Nevertheless, in time variant systems where the controller parameters should be adjusted according to variations in system dynamics, achieving good control performance is difficult. Designing good performance adaptive PID controllers have been a challenging issue in recent years. In adaptive PID controllers, controller parameters should be tuned according to changes in system dynamics. Different structures have been introduced for adaptive PID controller. Three main categories of such structures include conventional adaptive PID controllers [4]-[6], fuzzy adaptive PID controllers [7]-[9] and evolutionary based adaptive PID controllers [10]-[12]. Conventional adaptive controllers exhibit low performance behavior, fuzzy adaptive PID controllers require prior knowledge of the system to be adequately tuned and evolutionary based adaptive PID-controllers are not appropriate for fast dynamic systems because of their required training time. Neural network based adaptive controllers those employ supervised learning methods (ex. [10]) can be categorized in evolutionary adaptive controllers. As mentioned earlier, training process of these controllers need a period of time to be converged, which makes these controllers unsuitable for online instant applications. Unlike supervised learning approaches, there is no reference pattern in reinforcement learning methods. Reinforcement learning, which has origin in behaviorist psychology, adopts a test and verification method where the learning agent interacts with its environment and learns from the consequence of its actions [13][14]. As there is no reference to which convergence of algorithm is anticipated, the results of reinforcement learning can be instantly utilized, hence this learning approach can be employed in online and real time applications. Abstract: An adaptive PID controller is used to control of a two degrees of freedom under actuated manipulator. An actor-critic based reinforcement learning is employed for tuning of parameters of the adaptive PID controller. Reinforcement learning is an unsupervised scheme wherein no reference exists to which convergence of algorithm is anticipated. Thus, it is appropriate for real time applications. Controller structure and learning equations as well as update rules are provided. Simulations are performed in SIMULINK and performance of the controller is compared with NARMA-L2 controller. The results verified good performance of the controller in tracking and disturbance rejection tests.